System and method for representing a biological process via non-speech audio

ABSTRACT

A method of representing a biological process via non-speech audio and system for carrying out same are provided. The method is effected by extracting a sequence of time-related biological events from the biological process and transforming the sequence of time-related biological events into rhythm and/or melody representative of the sequence of time-related biological events thereby representing the biological process via non-speech audio.

RELATED APPLICATION/S

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 63/075,336 filed on Sep. 8, 2020, the contents ofwhich are incorporated herein by reference in their entirety.

FIELD AND BACKGROUND OF THE INVENTION

The present invention relates to a system and method for representing abiological process via non-speech audio (e.g., music). Embodiments ofthe present invention relate to sonification of an embryonic developmentvideo to generate a user audible rhythm and/or melody that can be usedto identify/verify embryos suitable for implantation.

In Vitro Fertilization (IVF) has been used to treat infertility problemssuccessfully since 1978. Despite on-going research it is still acomplicated procedure with a success rate of only 20% using the bestavailable resources.

IVF is an expensive procedure that is psychologically traumatic for apatient and as such, identifying recipients for whom IVF is unlikely tobe successful prior to treatment, or embryos most suitable forimplantation can reduce costs associated with an IVF procedure and thediscomfort such a procedure causes the patient.

The embryo selection step of an IVF procedure is crucial to implantationsuccess. Selection is typically carried out manually via microscopicscreening of embryos throughout their development cycle. More recently,time-lapse video microscopy has enabled prolonged time lapse capture ofmicroscopy images that can be analyzed automatically or manually todetermine suitability for implantation.

While video-based screening approaches increase the likelihood ofsuccessful implantation especially when manually carried out byexperienced technicians, additional screening tools are needed in orderto increase the likelihood of successful implantation.

While reducing the present invention to practice, the present inventorshave devised an approach for screening biological processes such asembryo development. The present approach can supplement manual orautomated video-based screening and provides a technician with anadditional layer of information that can be used in a decision makingprocess.

SUMMARY OF THE INVENTION

According to one aspect of the present invention there is provided amethod of representing a biological process via non-speech audiocomprising: extracting a sequence of time-related biological events fromthe biological process; and transforming the sequence of time-relatedbiological events into musical sonification representative of thesequence of time-related biological events thereby representing thebiological process via non-speech audio.

According to embodiments of the present invention, the sequence oftime-related biological events is extracted from a video or time lapsecapture of the biological process.

According to embodiments of the present invention, the biologicalprocess is embryonic development.

According to embodiments of the present invention, the sequence oftime-related biological events includes cellular division, growth and/ordifferentiation.

According to embodiments of the present invention, the sequence oftime-related biological events includes changes to a subcellularstructure.

According to embodiments of the present invention, the subcellularstructure is a nucleus, a pronucleus, cytoplasm or a cytoskeleton.

According to embodiments of the present invention, the rhythm ispercussion rhythm.

According to embodiments of the present invention, the method furthercomprises combining the rhythm and/or melody representative of thetime-related biological events to the video or time lapse capture of thebiological process.

According to embodiments of the present invention, the rhythm and/ormelody representative of the time-related biological events of a normalbiological process differs from that of an abnormal biological process.

According to embodiments of the present invention, the rhythm and/ormelody representative of the time-related biological events of a normalbiological process is more rhythmic and/or melodic than that of anabnormal biological process.

According to embodiments of the present invention, the sequence oftime-related biological events is extracted from a video or time lapsecapture of the biological process using image recognition software.

According to embodiments of the present invention, the rhythm and/or themelody is analyzed for changes over time.

According to embodiments of the present invention, the rhythm and/or themelody is analyzed using a signal-processing algorithm.

According to embodiments of the present invention, the signal-processingalgorithm extracts human-perceivable and quantifiable high-level musicalinformation.

According to embodiments of the present invention, the signal-processingalgorithm extracts a rhythmic periodicity of the rhythm and/or melody.

According to embodiments of the present invention, the signal-processingalgorithm measures a self-similarity of the rhythm and/or melody.

According to another aspect of the present invention there is provided asystem for representing a biological process via non-speech audiocomprising a computational unit configured for extracting a sequence oftime-related biological events from the biological process; andtransforming the sequence of time-related biological events into rhythmand/or melody representative of the time-related biological eventsthereby representing the biological process via non-speech audio.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Although methods and materialssimilar or equivalent to those described herein can be used in thepractice or testing of the present invention, suitable methods andmaterials are described below. In case of conflict, the patentspecification, including definitions, will control. In addition, thematerials, methods, and examples are illustrative only and not intendedto be limiting.

Implementation of the method and system of the present inventioninvolves performing or completing selected tasks or steps manually,automatically, or a combination thereof. Moreover, according to actualinstrumentation and equipment of preferred embodiments of the method andsystem of the present invention, several selected steps could beimplemented by hardware or by software on any operating system of anyfirmware or a combination thereof. For example, as hardware, selectedsteps of the invention could be implemented as a chip or a circuit. Assoftware, selected steps of the invention could be implemented as aplurality of software instructions being executed by a computer usingany suitable operating system. In any case, selected steps of the methodand system of the invention could be described as being performed by adata processor, such as a computing platform for executing a pluralityof instructions.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

The invention is herein described, by way of example only, withreference to the accompanying drawings. With specific reference now tothe drawings in detail, it is stressed that the particulars shown are byway of example and for purposes of illustrative discussion of thepreferred embodiments of the present invention only, and are presentedin the cause of providing what is believed to be the most useful andreadily understood description of the principles and conceptual aspectsof the invention. In this regard, no attempt is made to show structuraldetails of the invention in more detail than is necessary for afundamental understanding of the invention, the description taken withthe drawings making apparent to those skilled in the art how the severalforms of the invention may be embodied in practice.

In the drawings:

FIG. 1 schematically illustrates one embodiment of the present system.

FIG. 2 is an image showing mapping of cell division events (t events)for the major scale.

FIG. 3 illustrates a self-similarity matrix analysis of musicalstructure.

FIGS. 4A-B, 5A-B, 6A-B illustrate rhythm/melody audio tracks alignedwith several images representing a portion of a time lapse videodepicting development of a successfully implanted (FIG. 4A, 5A, 6A) anda non-successfully implanted (FIG. 4B, 5B, 6B) embryo.

DESCRIPTION OF SPECIFIC EMBODIMENTS OF THE INVENTION

The present invention is of a system which can be used to represent abiological process via rhythm and/or melody. Specifically, the presentinvention can be used to represent embryonic development in a mannerwhich enables a technician to acoustically detect, which embryos aremore likely to be successfully implanted.

The principles and operation of the present invention may be betterunderstood with reference to the drawings and accompanying descriptions.

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not limited in its applicationto the details set forth in the following description or exemplified bythe Examples. The invention is capable of other embodiments or of beingpracticed or carried out in various ways. Also, it is to be understoodthat the phraseology and terminology employed herein is for the purposeof description and should not be regarded as limiting.

Embryo selection in IVF procedures is a crucial step to implantationsuccess. Although advancements have been made in image capture ofdeveloping embryos and automatic analysis of captured images, thesuccess of a procedure still relies on the skill and experience of thetechnician. In a previously filed application, the present inventorsdevised an automated system for identifying predictors of successful IVFimplantation from images (e.g., of time lapse videos) captured fromsuccessfully and non-successfully-implanted embryos.

Addressing the need for additional tools, the present inventors deviseda system that can provide another layer of information that harnessesthe acoustic assessment of a user or automated system to substantiallyenhance an embryo screening procedure prior to implantation.

Thus, according to one aspect of the present invention there is provideda method of representing a biological process via non-speech audio.

The method is carried out by extracting a sequence of time-relatedbiological events from the biological process and transforming thissequence into a representative rhythm and/or melody (also termed hereinas “musical sonification”).

The biological process can be any process that includes a sequence ofbiological events over time. Examples of such processes include, but arenot limited to, disease progression, tumor growth, organ development,brain activity, heartbeat patterns, blood pressure changes, respiratorysystem and embryonic development. The biological events can be any macroor micro changes to a subcellular structure, a cell or a collection ofcells (e.g., tissue/organ) that can be observed. Examples of biologicalevents include, but are not limited to, cell division, cell growth,tumor growth or shrinkage, tissue/organ growth, migration or changes insubcellular structures such as nuclei, pronuclei, cytoplasm,cytoskeleton, endoplasmic reticulum and the like. These sequences can bemonitored in real time or, if their pace is slower or faster, using timecompression or expansion of the series of events.

The time-related biological events can be extracted from a video (e.g.,time lapse video capture of a microscopic scan) of the biologicalprocess via, for example, an image recognition software. The imagerecognition software can automatically identify and tag the biologicalevents with a time stamp and an identifier. For example, in the case ofcell division, image analysis implemented by means such as aconvolutional neural network can identify a cell division event and tagit with a time stamp and a ‘cell division’ identifier. Such an event canthen be used to generate a specific portion of the rhythm, melody,harmony and/or timbre as is further described hereinbelow.

Once the entire biological process (e.g., IVF embryo development frominsemination to transfer into the womb) is sonified, musicalsonification can be used to assess the nature of the biological process.For example, in the case of development (e.g., embryo), the audio trackcan be used to assess whether the development is normal or not. Theaudio track can also be aligned to, and combined with the video or timelapse capture of the biological process to provide an additional layerof information to a technician analyzing the video.

As is further described hereunder, musical sonification representativeof time-related biological events of a normal biological process differsfrom that of an abnormal biological process.

For example, a normal biological process can be perceived as morerhythmic than that of an abnormal biological process. More rhythmic inthis context can mean that the rhythm is more regular; e.g., that itconsists of rhythmic events that mainly fall on a beat. Less rhythmicpatterns consist of more events that occur between beats also known assyncopation. Less rhythmic patterns might also consist of suchirregularly spaced events that no beat is detectable at all. Complexprocesses such as cell division may result in multiple concurrentpatterns, where any part of the overall sequence may or may not berhythmic. For example, if some cells divide normally and someabnormally, the musical equivalent could be a beat with jarring noisesuperimposed.

As is mentioned hereinabove, the musical sonification can be analyzed bya user (e.g., technician) or by an automated system. In the case of theformer, the various constituent aspects of the musical output can bedesigned to intuitively convey “correctness” of a given input. Themusical aspects used can include multiple musical aspects such asrhythm, melody, timbre, and harmony. Rhythm, as described above, canconvey correctness intuitively via the level of syncopation; a moresyncopated (or less regular) rhythm will sound more chaotic and thusintuitively give the impression to the listener that the input may beless than ideal (less correct).

Melody can be used to convey correctness by relying on common patternsfound throughout western tonality. For example, individuals familiarwith western music will intuitively hear melodies that strongly suggesta diatonic major key as happy, positive, or good and will alternativelyhear melodies that do not suggest any diatonic tonal center as strange,uncomfortable, or unpredictable. This way, one can vary how strongly themelodies suggest a major diatonic center to convey to the listener howfit a given input is.

Timbre is a term that can convey the tone, or “color” of the musicalsounds. For example, one might describe a trumpet as having a harshertimbre than a French horn. In the present system, timbre is varied byintroducing more noise and distortion to the output in order to producea less pleasant tone. Sounds with more noisiness and distortions will beintuitively heard as less compelling by the listener.

Harmony is used in conjunction with the melodic aspects of thesonification to convey “correctness”. More successful inputs willproduce output that contains a drone tone that enforces the diatoniccenter suggested by the melodic content, whereas less successfulexamples will contain drone tones that are heavily distorted and areless related to a tonal musical center. When the drone tone is unrelatedto the melodic content, the output will be heard as dissonant tolisteners with experience with western music, and can convey a strongsense of incorrectness.

In the case of automated analysis, the musical sonification is generatedbased on a signal-processing algorithm that extracts human-perceivableand quantifiable high-level musical information (e.g., rhythmicperiodicity of the rhythm and/or melody) and can measure self-similarityof the rhythm and/or melody.

Self-similarity is used to detect patterns within a given input. So, amore rhythmic pattern for instance, where events occur at moreregularly-spaced intervals, will be more self-similar than a patternwith more varied or chaotic spacing. This automated approach can be usedto examine large amounts of sonification data without the need toindividually listen to and evaluate each example.

The human auditory cortex is highly tuned to detect patterns in temporalinput. For example, the human brain can notice time difference betweentwo audio stimuli that are separated by as little as 10 milliseconds.Such subtle time differences cannot be identified through visualization.Therefore, musical sonification bears can better represent patterns intime based events (such as embryonic cell division sequences) incomparison to visual based representation.

Referring now to the drawings, FIG. 1 illustrates one embodiment of asystem for representing a biological process via non-speech audio (alsoreferred to herein as sonification) which is referred to herein assystem 10.

System 10 is described hereunder in the context of embryo development,it will be understood however, that system 10 can also be used to assessalternative biological processes.

System 10 includes a computing platform 16 configured for obtaining andoptionally storing a sequence of time-stamped images trackingdevelopment of a pre-implantation embryo. System 10 can be incommunication (e.g. through cloud 13 or wire connection) with a datastorage device 12 storing the images (time lapse) retrieved from animage capture device 14 (e.g., microscope with camera) or it can be indirect communication with image capture device 14 through a wired orwireless (e.g., cloud 13) connection.

The time-stamped images can be derived from a video capture, a timelapse capture or a still capture at various time intervals. For example,the images can be derived from stored image data that includes timelapse/video/still images obtained from an embryo.

In any case, the time-stamped images represent a time period of −150hours in embryo development (the time from fertilization to 6 days postfertilization) from 0 hours until ˜150 hours and can be spaced apart at1 second to 20 minute intervals.

The following represents typical time points of biological events indevelopment of a fertilized oocyte to implantation-ready embryo that canbe included in the time stamped sequences of images utilized by thepresent invention for sonification of embryonic development.

t0: The time at which insemination occurs in conventional IVF. ForICSI/IMSI, where the time-lapse monitoring system and practice allows,the time of the sperm injection may be recorded, per oocyte butotherwise, it is the mid-time point from when injection begins and endsfor that patient's cohort of oocytes. This time point can be used as astart time.

tPB2: The time at which the second polar body (PB2) is extruded. This isannotated at the first frame in which PB2 appears completely detachedfrom the oolemma. The extrusion of the second polar body can be obscureddepending on the position of the oocyte in the well or by cumulus cellsin routine IVF insemination.

tPN: The time at which fertilization status is confirmed. It isrecommended to annotate fertilization immediately before fading ofpronuclei (tPNf) hence coinciding to tZ (time of pronuclear scoring),since no further observational dynamic changes are expected to occur.Appearance of individual pronuclei may be further annotated as tPNna(′n′ for individual pronuclei in the order of appearance: ‘a’): e.g.tPN1 a, tPN2 a, tPN3 a the initial time at which the first, second,third, etc. pronuclei become visible.

tPNf: The time when both (or the last) PN disappear. This annotation ismade at the first frame whereby the embryo is still at the 1-cell stagebut pronuclei can no longer be visualized. Pronuclear fading may befurther recorded according to individual pronuclei, tPN1 f, tPN2 f, etc.to denote the time at which the first, second or additional pronucleifade (i.e. similar to annotation of their appearances).

tZ: The time of time-lapse PN assessment. PN are dynamic structures;they move and their morphology can change between tPNa and tPNf(Azzarello et al., 2012). It has recently been reported that themovement of the pronuclei within the cytoplasm and fading of nuclearmembranes may be indicative of subsequent blastocyst developmentpotential and hence a novel parameter providing an early indication ofthe embryo's developmental potential (Wirka et al., 2013). Changes inpronuclear appearance and position may coincide with movement of thenucleolar precursor bodies (NPBs) inside pronuclei, allowingdifferential PN scoring to be deduced. The time-lapse user grouprecommends annotation of PN scoring, if required, at the last framebefore the pronuclei disappear (i.e. tPNf) because the alteration inpronuclear morphology has been completed.

t2: The time of the first cell cleavage, or mitosis. t2 is the firstframe at which the two blastomeres are completely separated byindividual cell membranes.

t3: The first observation of three discrete cells. The three cells stagemarks initiation of the second round of cleavage.

tn: The first time these numbers of discrete cells are observed (untilcompaction of blastomeres prevents visualization of individual cells).

tSC: The first frame in which evidence of compaction is present; theinitial frame that any (two) cells start to compact is observed.

tMf/p: This marks the end of the compaction process; when observablecompaction is complete. The morula may be fully or partially compacted,where f is full and p is partial; the morula has excluded material. Thedegree and time of compaction has been reported to be associated withblastocyst formation and quality (Ivec et al., Fertility and sterility,Volume 96, Issue 6, December 2011, Pages 1473-1478.e2 2011).

Any of the above biological events can be used in creating the audiotrack (rhythm/melody) of the present invention.

Once an audio track of embryonic development is generated, the presentsystem can play the track to a user along with, or separately from, thevideo. A listener will then be able to play the audio, and hear thevarious musical aspects described above (melody, rhythm, timbre, andharmony), to be able to make an intuitive assessment of the output.

FIGS. 4A-B, 5A-B and 6A-B illustrate examples of musical sonification ofa successfully implantation (FIG. 4A, 5A, 6A) and an unsuccessfulimplantation (FIG. 4B, 5B, 6B). The images on top represent stages inembryo development: the first, second and third mitosis events (divisionof a cell into two), the images on the bottom represent sonification.After fertilization, the egg goes through a process which ends in ablastocyst, a mass of cells ready for implantation in the womb. Mitosisexhibits a temporal pattern specific to the embryo so that, for example,if the first cell divides early—relative to the average of embryos—thenthe second and third cells are expected to also divide early. Deviationsfrom the expected pattern, as is shown in FIGS. 4B, 5B and 6B correlatewith poor embryo outcomes.

The bottom panels of FIGS. 4A-6B illustrate different musical elementsof the sonification that are generated in accordance with whetherdivision is of a successfully implanted embryo or one that was notsuccessfully implanted. Each of these figures shows three divisions ofthe candidate embryo (t2, t3, t4), which are represented as the threearrows between the four images of the embryo at the upper portion ofeach figure. FIG. 4A shows a resultant melody for a successfullyimplanted embryo. Three division events (t2, t3, t4) that are connectedwith thin lines are illustrated so as to show how these division eventsdirectly correspond with the generated melodic events (each divisionevent directly triggers a note to play). The staff in this example istreble clef, but this was left out so as to not overcrowd the diagram.The notes being generated in this example strongly suggest a diatonicmajor tonality, since the three notes spell out a C major chord. Inwestern music, a major chord is often associated with positivity ingeneral, so its use here corresponds nicely with the success of thisembryo.

FIG. 4B is an example of an unsuccessfully implanted embryo and theresulting melody generated by the present invention. Unlike the melodyin 4A, the melody in 4B does not strongly suggest a diatonic tonality.The first two notes in the 4B melody are a distance apart that is knownas a “tritone”. In western music, a tritone is often considered a harshinterval that is most often used to indicate uneasiness of some sort, sothe presence of this interval and lack of implied tonality in thismelody reinforce to the listener that this embryo is not a successfulone.

FIG. 5A shows a rhythm that is generated for an embryo that wassuccessfully implanted. The rhythm generated here is essentially a basicrock and roll beat. In this rhythm, each downbeat is stressed by a kickdrum hit—represented as the low a in this notation (these examples, likeexamples 4A and 4B are also written in treble clef)—with the weak beatsor upbeats being stressed by a snare hit (middle c), so each drum hitoccurs directly on a beat. There is a straight eighth note high-hatcymbal pattern happening along with these drum hits that helps give therhythm a strong and apparent sense of beat. The regularity of thisrhythm should convey a sense of “expectedness” to the listener; thestrong sense of beat and typical beat-stressing pattern of the rhythmcorrespond well with the example embryo's success.

FIG. 5B is an unsuccessfully implanted embryo and its resultant rhythm,unlike the rhythm of FIG. 5A, most drum beats do not occur on the beat.When rhythms have events that occur off the beats, this is known assyncopation. Syncopation does not necessarily convey a negative feelingto the user like, for example, an atonal melody, however these rhythmsare often heard as more exotic and possibly even strange. Only one drumbeat actually aligns with a beat (the second kick drum hit), howeverthis doesn't wind up giving this rhythm any more perceived regularity.Since the rest of the elements are so irregular, a listener may hearthis rhythm as being almost, if not entirely, arrhythmic. This irregularand highly syncopated rhythm clearly conveys that this embryo is notsuccessful.

FIGS. 6A and 6B are spectrograms of a generated melody of thesonication. The spectrograms are used to illustrate how the soundelements of the melody, beyond just its constituent notes, are generatedso as to reflect the success or failure of a candidate embryo.

FIG. 6A shows the timbral component of a sonification performed for asuccessfully implanted embryo. A spectrogram is used herein toillustrate this timbral component. A spectrogram shows us the frequencycontent of a given sound and demonstrates nicely divided parallel linesin the Example of FIG. 6A. The parallel lines are segmented into threesections, each of different vertical offset; this simply represents thatthere are three separate notes played. Much like examples 4A and 4B,direct mapping between the division events (t2, t3, and t4, representedas the three arrows in the upper portion of the figure), to the threeseparate notes visible in the spectrogram is shown. In this case,importance is placed on how distinct each of the parallel lines in thespectrogram is; this implies that these notes will have a clear pitch.Thus, a successful embryo will have a timbre that is very clear and doesnot include any noisy sonic components.

FIG. 6B shows the spectral (or timbral) component of a resultantsonification of a failed embryo. The same parallel line structure as inFIG. 6A is shown, but it is clear that the lines are far less distinct;this means that the tones/pitches of this example would be far lessclear and would sound a fair amount more noisy than those in FIG. 6A.The noisiness and lack of clear tones conveys a more negative impressionto the user, since a more noisy sound will typically be perceived asstrange or even unpleasant than an equivalent less noisy sound.

As is mentioned hereinabove, the present invention can also be suitablefor representing disease progression, tumor growth, organ development,brain activity, heartbeat patterns, blood pressure changes, respiratorysystem as well as other pathologies, conditions and disorders.

As used herein the term “about” refers to ±10%.

Additional objects, advantages, and novel features of the presentinvention will become apparent to one ordinarily skilled in the art uponexamination of the following examples, which are not intended to belimiting.

Example

Reference is now made to the following example, which together with theabove descriptions, illustrate the invention in a non-limiting fashion

Sonification of Embryonic Development

A system for sonifying time-lapse videos of fertilized embryos wasdeveloped and tested. The system enables to explore embryonicdevelopment in the sonic domain. This system has three main proposeduses (i) detection of embryos that are more likely to succeed, (ii)provide an aesthetic fingerprint for an embryo for artistic purposes toameliorate the process mothers go through while undergoing in vitrofertilization and (iii) provide researchers with new tools andtechniques found in music and sound research that can be applied toscreening of biological processes.

The sonification system utilizes percussion, drum loops, melodies,drones and the like to convert time-stamped biological events present intime lapse videos of embryo development to musical sonification audibleby a user. Each video is analyzed by a data analysis system, producingvarious time-series data points. The data points used in this study arelabelled t2, t3, t4, t5, t6, t7, t8, and t9 and correspond to celldivision event (t3 for example, is the first frame at which at least 3cells appear).

The percussion feature of the sonification is the first and most basicfeature. Each of the eight t events trigger a unique percussion sample.This generates a unique percussion track for each embryo video. The useris also able to choose one or more percussion sounds to be mapped toeach t event. The percussion sounds can include various kick drum, handdrum, and cymbal samples.

The drum loop feature allows for a drum loop to be played in time withthe time-lapse video, such that the drum loop repeats seamlessly. Thisfeature is added to add a background rhythmic context upon which theother sonification elements can be stacked. The idea is that thisbackground rhythmic context may help users interpret the sonification ina more rhythmic sense; allowing users to detect subtle time differencesthat might not be detectable outside of a non-musical context. The otheruse of the drum loop is to provide an aesthetic backbone for thesonification to increase overall musicality.

The melody feature of the sonification is much like the percussionfeature except that the t events are mapped to different notes of amusical scale rather than to different percussion samples. There are twomodes that the melody mode operates under, custom scale mode and a fixedscale mode. The fixed scale mode is the more simple of the two modes. Inthe fixed scale mode, each t event (2 through 9) is mapped to thispattern of scale degrees: 1, 3, 8, 6, 4, 2, 5, 7. The system can have 7scales to choose from: major, minor, pentatonic, persian, nahawand,bayati, and jiharkah. Nahawand, bayati, and jiharkah are scales takenfrom the Arabic maqam tradition. The pattern for scale degrees wasmainly chosen for the way grouped events form a pleasant implied chordprogression when a western diatonic scale (major or minor) is chosen.

FIG. 2 illustrates an example of mapping oft events for the major scale.Here, the implied chord progression is I-ii-V, a very common progressionin wester harmony. If the chosen scale does not have at least 7 distinctnotes (the pentatonic scale for instance), the scale is continued beyondan octave until there are enough notes to create the scale degreepattern. This means that, this pattern does not create a nice impliedchord progression in all cases (ie the pentatonic scale). However, thescale degree pattern also makes sure that two consecutive t eventstrigger notes that are not consecutive in the desired scale; this willgenerally avoid dissonant-sounding chords in cases where two t eventsoccur close together. In the more complex custom scale mode, therelative spacing of the t events determine the frequencies of each notein the scale. This means that each embryo will have a unique scaleassociated with it. The midi pitch of the ith degree of the custom scaleis represented by the equation:

${{Pitch}(i)} = {60 + \frac{t_{i} - t_{2}}{t_{9} - t_{2}}}$

This midi pitch can then be translated into the desired frequency ofeach note in the custom scale.

The first variation of the drone provides an aesthetic backdrop to theother sonification features and is based on overall visual activity.Both this variant of the drone and the second variant are different fromthe other sonification features in that they analyze the video directly,whereas the other sonification features rely on higher-level labeleddata (the t events). The analysis for this feature is quite primitiveand low level. An average pixel difference between two consecutive videoframes is determined and fed into a buffer in order to use a runningmedian filter on the previous 11 values to produce a filtered stream ofdata that corresponds to the amount of movement occurring in the video.This stream of data is then mapped onto various synthesizer parametersin order to produce a droning sound that becomes brighter and moreactive in its tone as the video becomes more active in terms ofmovement. This drone is centered on a C note, which was chosen as thecommon starting note of all of the scales present in the melody feature,making the drone musically complimentary to the rest of the sonificationsoundscape.

The second variant of the drone, much like the first variant, analyzesthe video to capture movement by using the Horn-Schunk technique tocalculate optical flow. Optical flow is typically used in videocompression algorithms; it attempts to locate only pixels that move fromframe to frame and then encode only that pixel movement instead of thewhole video frame, thus compressing the video file size. This study didnot use optical flow to compress the videos, but rather to determinemovement in the videos. The optical flow data was broken into fourframes, corresponding to all the pixel movement occurring in eachdirection (left, right, up, & down). Each of these frames was thenaveraged and filtering applied in a manner similar to that of the firstvariant of the drone producing four streams of data rather than justone. The basic or the drone is created by mapping these four datastreams onto the operator frequencies of a four-operator fm synthesizer.The four streams of data were then used to spatialize the drone. Thespatialization of this drone is the only aspect of the sonification thatrelies on stereo audio. The basic idea of the mapping for spatializationpurposes is to make the left or right channel more or less active basedon the amount of optical flow to the left or right. For furthermodulation of the drone, there is another idea at play in this mappingthat requires some description of the input videos.

The videos show cells growing and dividing it is therefore expected thatmovement in the video would be similar in the four basic directions,since the cells appear as circles in the videos. However, when cellsdivide, there is less of an expectation that movement will be similar inall directions. In order to represent such asymmetry in movement, thesimilarity between the four streams of optical flow data was mapped tovarious parameters of the drone synthesizer to render the drone moredistorted and active when similarity is lower. Since this variant of thedrone is a constantly changing pitch, it does not necessarily complimentthe melody feature but can be made more musical.

Data Analysis

Data analysis utilized a self-similarity matrix (SSM) to performstructural analysis of the sonification audio assuming that a techniqueoften used in musical structural analysis—the SSM-might be applicable tothis domain.

SSM are used for musical structural analysis, the basic idea behind theSSM is to show how similar each point throughout an input audio sampleis similar to each other point. An Example of a computed SSMs is shownin FIG. 3 in which brighter pixels correspond to higher degrees ofsimilarity.

In order to calculate the SSM, the sonification audio was broken intowindows 25 ms in length sampled every 100 ms by default. The interfaceallows these values to be changed by the user. The Mel FrequencyCepstral Coefficients (MFCCs) was then calculated for each of thesewindows and then each feature vector was grouped with the 9 featurevectors following it, creating groupings of 10 feature vector. Thesegroupings of 10 feature vectors are then compared with each othergrouping by summing the pairwise dot products of each group, producing asingle similarity score. So, given feature vectors F[1 . . . n], andgroup size S (10 by default) an (n-S-1) by (n-S-1) matrix is computedwhere each cell in the matrix—for coordinates x and y where both x andwhy range between 1 and (n-S-1)—is represented by the equation:

|cell(x,y)=Σ_(i=1) ^(S) F[x+i−1]·F[y+i−1].

The dataset includes 2248 time lapse videos, each with correspondingpre-calculated time-series data. One piece of data associated with eachembryo video is a flag called “beta_pregnancy_test”, which indicateswhether an embryo was successfully implanted or not. The dataset wasdivided based on this flag to 1197 failed embryos and 1051 successfulembryos.

To examine if there is a significant difference in overall similaritybetween the successful embryos and the failed embryos a batch outputfeature was added to the sonification software. Batch output allows auser to specify a folder containing multiple videos files for which theuser wishes to compute SSMs. Once a folder is selected, the systemautomatically sonifies each file, then it computes the SMM for eachsonification. Each SSM is stored as a CSV file, rather than an image, sothat data can be precisely saved and reloaded for further analysis. TheSSMs were batch calculated for all of the successful and failed embryos.Since the sonifications typically contained some amount of silence atthe beginning or end, a threshold to remove points of low similarity atthe edges of the SSMs was set. The average similarity for the entirematrix was then calculated and the average similarity score for all ofthe successful and failed embryos was calculated. The means of these twosets was then compared and the results showed that successful embryoshave higher average similarity scores in general than failed embryos.Running a t-test produces a t statistic of 4.56 with a p value of5.80e⁻⁶.

As is mentioned hereinabove, the data used in sonification includes asequence of time events labeled t2 through t9, representing the times atwhich cell divisions occur. Since sonification represents a mapping ofthese events to particular sounds, embryo division data was used tocompute SSMs without relying on the sonification audio. This approachrelies on techniques typically used in musical structural analysis, andwas directly inspired by the use of sonification. Computing these SSMswas far simpler than computing the SSMs for the sonified audio. In orderto compute each SSM, the series of t2 to t8 was used as the basic input.The similarity (simple euclidean distance) of each rotation of the inputvercor to all possible rotations was then computed producing an 8×8matrix. For illustration, the first rotation of the input vector wouldlook like [t3,t4,t5,t6,t7,t8,t2]; notice t2 is now at the end of thevector after the first rotation. Using this technique on the entiredataset enabled assignment of an average similarity value to eachembryo. Results showed that as with the previous approach (computingSSMs from the sonification), successful embryos have higher averagesimilarity scores in general than failed embryos. Running a t-testproduces a t statistic of −3.55 with a p value of 0.00040. The sign ofthe t statistic is negative for this experiment since given the waysimilarity is computed in this case, smaller values imply moresimilarity. So this result matches the result from the previousexperiment.

The present study then tested if these SSMs could be used as a featureto be fed to a machine learning model used to classify whether acandidate embryo was successful or not by using the 8×8 SSMs generatedfrom looking at the event data directly. The study first balanced thedataset so that there are equal numbers of failed embryos and successfulembryos. The data was then divided into 1686 examples used to train themodel and 562 examples to test the model; binary classification wasperformed by a support vector machine. Running the model on the testdata, resulted in an area under the curve of the region of convergencegraph (AUC-ROC) of 0.55.

From statistical analysis, it is clear that the rhythms generated bydivision events are significantly more self-similar for successfulembryos as opposed to failed embryos.

It is appreciated that certain features of the invention, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable subcombination.

Although the invention has been described in conjunction with specificembodiments thereof, it is evident that many alternatives, modificationsand variations will be apparent to those skilled in the art.Accordingly, it is intended to embrace all such alternatives,modifications and variations that fall within the spirit and broad scopeof the appended claims.

All publications, patents and patent applications mentioned in thisspecification are herein incorporated in their entirety by referenceinto the specification, to the same extent as if each individualpublication, patent or patent application was specifically andindividually indicated to be incorporated herein by reference. Inaddition, citation or identification of any reference in thisapplication shall not be construed as an admission that such referenceis available as prior art to the present invention. To the extent thatsection headings are used, they should not be construed as necessarilylimiting. In addition, any priority document(s) of this applicationis/are hereby incorporated herein by reference in its/their entirety.

What is claimed is:
 1. A method of representing a biological process vianon-speech audio comprising: (a) extracting a sequence of time-relatedbiological events from the biological process; and (b) transforming saidsequence of time-related biological events into sonificationrepresentative of said sequence of time-related biological eventsthereby representing the biological process via non-speech audio.
 2. Themethod of claim 1, wherein said sequence of time-related biologicalevents is extracted from a video or time lapse capture of the biologicalprocess.
 3. The method of claim 1, wherein the biological process isembryonic development.
 4. The method of claim 3, wherein said sequenceof time-related biological events includes cellular division, growthand/or differentiation.
 5. The method of claim 3, wherein said sequenceof time-related biological events includes changes to a subcellularstructure.
 6. The method of claim 5, wherein said subcellular structureis a nucleus, a pronucleus, cytoplasm or a cytoskeleton.
 7. The methodof claim 1, wherein said rhythm is percussion rhythm.
 8. The method ofclaim 2, further comprising combining said rhythm and/or melodyrepresentative of said time-related biological events to said video ortime lapse capture of the biological process.
 9. The method of claim 1,wherein said rhythm and/or melody representative of said time-relatedbiological events of a normal biological process differs from that of anabnormal biological process.
 10. The method of claim 9, wherein saidrhythm and/or melody representative of said time-related biologicalevents of a normal biological process is more rhythmic and/or melodicthan that of an abnormal biological process.
 11. The method of claim 2,wherein said sequence of time-related biological events is extractedfrom a video or time lapse capture of the biological process using imagerecognition software.
 12. The method of claim 1, wherein said rhythmand/or said melody is analyzed for changes over time.
 13. The method ofclaim 12, wherein said rhythm and/or said melody is analyzed using asignal-processing algorithm.
 14. The method of claim 13, wherein saidsignal-processing algorithm extracts human-perceivable and quantifiablehigh-level musical information.
 15. The method of claim 13, wherein saidsignal-processing algorithm extracts a rhythmic periodicity of saidrhythm and/or melody.
 16. The method of claim 13, wherein saidsignal-processing algorithm measures a self-similarity of said rhythmand/or melody.
 17. A system for representing a biological process vianon-speech audio comprising a computational unit configured for: (a)extracting a sequence of time-related biological events from thebiological process; and (b) transforming said sequence of time-relatedbiological events into rhythm and/or melody representative of saidtime-related biological events thereby representing the biologicalprocess via non-speech audio.